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ABSTRACT 

A framework of online adaptive statistical compressed sensing is in- 
troduced for signals following a mixture model. The scheme first 
ms^es non-adaptive measurements, from which an online decoding 
^rriheme estimates the model selection. As soon as a candidate model 
as been selected, an optimal sensing scheme for the selected model 
"continues to apply. The final signal reconstruction is calculated from 
ensemble of both the non-adaptive and the adaptive measure- 
lents. For signals generated from a Gaussian mixture model, the 
iline adaptive sensing algorithm is given and its performance is an- 
On both synthetic and real image data, the proposed adaptive 
O^heme considerably reduces the average reconstruction error with 
respect to standard statistical compressed sensing that uses fully ran- 
Kdom measurements, at a marginally increased computational com- 
^exity. 

• Index Terms — Statistical compressed sensing, adaptive sensing, 
Qaussian mixture models, model selection. 

^ 1. INTRODUCTION 

febmpressed sensing (CS) aims at achieving accurate signal recon- 
^jQ-uction while sampling signals at a low sampling rate, typically far 
^gialler than that of Nyquist. Let x G be a signal of interest, 
G M^>^^ a non-adaptive sensing matrix {encoder), consisting of 
Af ^ N measurements, y = <J>x G a measured signal, and A a de- 
coder used to reconstruct x from <l>x. CS develops encoder-decoder 
^ p^ irs (<J>,A) such that a small reconstruction error ||x — A(<l>x)||x, 
r^^iere || • ||x is a norm, can be achieved. 

Assuming a sparse signal model, i.e., the signal can be accurately 
. ^cqpresented in a dictionary with a few non-zero coefficients, the CS 
^^j^eory has shown that using random sensing matrices such as Gaus- 
or Bernoulli matrix <l> with M = ^(k\og(N /k)) measurements, 
and an l\ minimization or a greedy matching pursuit decoder A pro- 
moting sparsity, with high probability accurate signal reconstruc- 
tion is possible: The obtained approximation error ||x — A(<l>x) ||x 
is tightly upper bounded by a constant times the best ^-term approx- 
imation error in the sparse representation |3 1. 

While conventional CS deals with one signal at a time, statistical 
compressed sensing (SCS) aims at efficiently sampling a collection 
of signals and having accurate reconstruction on average. Assuming 
that the signals x follow a distribution with probability density func- 
tion (pdf) /(x), SCS designs encoder-decoder pairs (<l>, A) so that the 
average error£||x — A(<J>x)||x = / ||x — A(<l>x)||x/(x)(ix is small |6|. 

For signals following a Gaussian distribution, it has been shown 
that with any sensing matrix of M = ^ measurements and the maxi- 
mum a posteriori (MAP) linear decoder, SCS leads to a mean squared 
error (MSE) £'||x — A(<J>x) |p upper bounded by a constant times the 
minimum MSE obtained with the ^-term linear approximation in the 
principal direction analysis (PCA) basis that is optimal for Gaussian 



signals f6l. In particular, the error bound is tight when Gaussian or 
Bernoulli random sensing matrix is used |6|. For signals generated 
from a Gaussian mixture model (GMM), i.e., there exist multiple 
Gaussian distributions and each signal is generated from one of them 
with an unknown index, GMMs giving more precise description of 
most real signals than single Gaussian models, a piecewise linear de- 
coder that calculates the signal reconstruction from each of the Gaus- 
sian models, and then selects the best one, has been introduced |[6||7l. 
Additional theoretical results on the Gaussian model selection accu- 
racy and overall reconstruction have been shown in 1 2 1 . 

SCS of GMM applies non-adaptive random sensing matrices be- 
cause for the signal being sensed, the Gaussian model from which 
the signal is generated is a priori unknown. If it were known, one 
would then prefer sensing along the principal directions in the ap- 
propriate Gaussian, which leads to the minimum MSE^More gen- 
erally speaking, assume that the signals are generated from a mixture 
model, and an optimal sensing scheme is associated with each of the 
underlying models (e.g., following 1 1 1). If for the current signal, its 
model were known before sensing, the optimal sensing scheme in 
that distribution would be preferred rather than using non-adaptive 
measurements. 

This paper follows this line of thoughts and introduces an on- 
line adaptive sensing framework for signals generated from a mix- 
ture model. The scheme imbeds an online model selection and a 
switch from non-adaptive to adaptive sensing. To sense a signal, 
non-adaptive measurements are first used, from which an online de- 
coding scheme calculates the model selection. As soon as a model 
has been selected, the optimal sensing scheme of the selected model 
then continues to apply. The final signal reconstruction is calculated 
from the ensemble of both the non-adaptive and the adaptive mea- 
surements. 

As an important example, this online adaptive sensing is here il- 
lustrated for signals following a GMM. Not only GMMs have been 
shown to lead to results in the ballpark of the state-of-the-art in vari- 
ous inverse problems for different types of real data ||4l|7l, theoretical 
results on statistical compressed sensing of GMM have also been re- 
cently given |51 [SI . 

Section [2] recalls the main results of SCS of GMM | 6 | based on 
which the online SCS of GMM will be developed. An algorithm 
for the online adaptive SCS of GMM is then given in Section [3] and 
its performance is analyzed and compared against standard SCS us- 
ing fully random sensing. In Section]?] the proposed online adaptive 
SCS is applied in real image data sensing, leading to considerably 
improved results with respect to standard SCS, at a marginally in- 
creased computational complexity. Concluding remarks and future 



^This optimal MSE sensing for Gaussians is easy to prove, see next Section, while 
the optimal sensing for other distributions has been recently elegantly developed in [TJ. 
The strategy here introduced can then be extended to mixtures beyond GMMs. 



work are discussed in Section [S] 



/ = arg max - ^ (log 1 + xjz^- 4^) , 



(5) 



2. STATISTICAL COMPRESSED SENSING 

2.1. Sensing of Gaussian Models 

2.1.1. Optimal Principal Direction Sensing 

Signals x G are assumed to follow a Gaussian distribution 
where jl and L are respectively its mean and covari- 
ance. Without loss of generality, the Gaussian mean is assumed 
zero, /i = 0, as one can always center the signal with respect to the 
mean. Principal Component Analysis gives the orthonormal PCA 
basis B that diagonalizes the covariance matrix L = BSB^, where 
S = diag(Ai, . . . , Aiv) is a diagonal matrix whose diagonal elements 
^1 > A2 > . . . > are the sorted eigenvalues |5 1. It is well known 
that for Gaussian signals a linear approximation in the PCA basis 
minimizes the mean squared error (MSE). Putting this in the signal 
sensing context, a sensing matrix 

*=[bi,...,bM]^GM«^^ (1) 

where is the ^-th principal direction of the Gaussian, i.e., the ^-th 
column in B, and a linear decoder 

A=[bl,...,bM]eR^><^ (2) 

minimize the MSE amongst all sensing matrices <I> G M^><^ and any 
decoder A: 

M N 

al^ min ^[||x- A(<Dx)|p] = ^[||x- ^ (x,b,)b,|p] = ^ A,, 
where cr^ denotes the minimum MSE. 

2.1.2. Statistical Compressed Sensing 

For Gaussian signals, it has been shown that any sensing matrix 
<I> G and the maximum a posteriori (MAP) linear decoder 

A = L<I>^(<I>Z<I>^)~^ lead to an MSE upper bounded by a constant 
times the minimum MSE O : 

Theorem 1 Assume x ~ ^(0, Z). Let ^beanMxN sensing matrix 
and A = Z<I>^(<I>Z<I>^)~^ the optimal and linear decoder Then 

£[||x-A(*x)||i]<Co4, (3) 

where the constant Cq is defined in 

The bound constant Cq in Theorem [T] can be obtained via Monte 
Carlo simulations. For Gaussian and Bernoulli matrices, a small 
Co ~ 4.5 has been shown, i.e., the error bound is tight I6i|. 

2.2. Sensing of Gaussian Mixture Models 

A single Gaussian distribution is often too simplistic for modeling 
real signals. Assuming multiple Gaussian distributions {^(0, ^j)}i<j<j 
and that each signal follows one of them with an unknown index, 
Gaussian mixture models (GMMs) provide more precise signal de- 
scriptions. 

As the Gaussian indices of the signals are unknown, the optimal 
principal sensing ([T]), ^ is impracticable. SCS applies instead non- 
adaptive random matrices for signal sensing and a piecewise linear 
decoder for reconstruction 1 6 1 . The piecewise linear decoder first cal- 
culates the linear MAP decoder using each of the Gaussian models, 

x^- = Aji^x) = Zj^^i^Zj^^)-^ (<J>x), VI < J < /, (4) 
and then selects the best model j that maximizes the log a-posteriori 
probability among all the models (Tl 



whose corresponding decoder Aj gives the final signal reconstruc- 
tion: , , , , 

A{^x)=Aj{^x). (6) 

The accuracy of the Gaussian model selection ^ and of the sig- 
nal reconstruction given by the piecewise linear decoder has been 
shown influenced by a number of factors, including the geometry of 
the Gaussian distributions in the GMM, the signal dimension, and 
the number of sensing measurements |6|. More accurate model se- 
lection and smaller reconstruction error is obtained as the Gaussians 
distributions are more "orthogonal" one another, as each of the Gaus- 
sians is more anisotropic, as the signals are in a higher dimension 
given that the energy of the signals are concentrated in the first few 
dimensions, and as the number of sensing measurements increases. 
Additional theoretical results on Gaussian model selection have been 
given in |2|. 

3. ONLINE ADAPTIVE STATISTICAL COMPRESSED 
SENSING 

SCS of GMM applies non-adaptive random sensing matrices be- 
cause for the signal to be sensed, the Gaussian model from which the 
signal is generated is a priori unknown. If it were known, one would 
then prefer sensing along the principal directions in the appropriate 
Gaussian, which leads to the minimum MSE. 

The online adaptive SCS improves the accuracy of SCS by first 
selecting online the Gaussian model, and then adapts the measure- 
ments as a function of the model selection. It starts by performing 
non-adaptive random measurements, based on which the piecewise 
linear decoder estimates online the Gaussian model for the signal be- 
ing sensed. As soon as the Gaussian model is selected, for the rest of 
the measurements it switches to the principal direction sensing in the 
selected Gaussian. As long as the online model selection is correct, 
the adaptive sensing along the principal directions in the appropriate 
Gaussian leads to a smaller MSE than applying fully random sensing. 

3.1. Algorithm 

Assume that M <N measurements are dedicated to sensing the sig- 
nal. The online SCS algorithm proceeds as follows. 

1. Random sensing. Sense the signal with a random matrix 
<I>f G M^''^ of ^ < M measurements. 

2. Online decoding and model selection. Decode online the sig- 
nal from <I>|^x using the piecewise linear decoder ^ and ([5]): 

^ AX*|x) = E,-<D^(*|EX*f VI < y < 7, 

(7) 

; = arg max - ^ (log \Lj \ + {y^^jY^]'^) ■ (8) 

3. Optimal sensing. Sense the signal with = [b{ , . . . , ^m-k] 
^{M-K)xN^ i.e., the first M — K first principal direction vectors 
in the j-th Gaussian selected online in ([S]). 

4. Decoding. Write <I>x = [(<I>f x)^, (<I>^_^x)^]^ G M^^'^ the 
concatenation of the signal measurements sensed in steps 1 
and 3. Decode the signal from <I>x with the piecewise linear 
decoder (|4]), ([5j, and ([6]). 

Contrary to the conventional CS and SCS that apply linear sens- 
ing, the sensing of the online adaptive SCS is nonlinear, as the princi- 
pal direction sensing matrix ^m-k in Step 3 depends on the Gaussian 



model selection estimated from the random measurements sensed in 
Step 1. 

The online adaptive sensing algorithm marginally increases the 
computational complexity with respect to standard SCS using fully 
random measurements. The sensing complexity is the same, but the 
online SCS has an additional online decoding step. The complexity 
of decoding ^ is dominated by the M x M matrix inversion, which 
requires floating-point operations (flops) |7|. With a GMM 

comprised of / Gaussian distributions, the online and the final de- 
coding steps are respectively calculated in JK^ /3 and flops. 
As K <M, the additional online decoding brings a marginal increase 
in computational complexity. 

Adjusting the number K of random measurements in the online 
SCS trades off between the online Gaussian model selection accu- 
racy in Step 2 and the signal reconstruction error in Step 4. The larger 
the K, the more random measurements are dedicated, and more ac- 
curate the online Gaussian model selection is in consequence (see Q 
for the exact bounds). Given the correct online Gaussian model se- 
lection, a smaller K leaves a bigger number M — K measurements 
along the principal directions of the appropriate Gaussian, which re- 
duces the signal reconstruction error. 

3.2. Performance Analysis 

To better understand the performance of the online adaptive SCS, 
let us analyze a GMM comprised of two Gaussian distributions 
^(0,Li) and ^(0,L2)- Assume without loss of generality that 
the signals follow the first Gaussian distribution x ~ ^(0,Zi). The 
MSE of the online SCS can be written as 

2 2 

E\\x-A{^x)f = '£l^ [ ^ ^ J\x-Ajj{^f\)fMx)dx (9) 

where /i(x) = (2;,)iv/2p^|i/2 ^^p (-^x^Z~4), / and j index re- 
spectively the Gaussian model selected online and at the final 
signal reconstruction, <I>1^^ = [(<l>|^)^, (<l>^_^)^]^ is the concate- 
nation of the random sensing matrix <$>^ of K measurements and 

the principal direction sensing matrix = [^{j - - - j^m-kV 

of M — K measurements in the Gaussian selected online, and 
Ajj = Lj(<J>l^V(<J>l^^Ej(<l>f y ^ includes 4 components: 

1. 7 = 1 and 7 = 1: Both the online decoding in Step 2 and the 
final decoding in Step 4 correctly select the Gaussian model 
for the signal. 

2. 7 = 1 and 7 = 2: The online decoding correctly selects the 
Gaussian model, whereas the final decoding incorrectly selects 
the Gaussian model. 

3. 7 = 2 and 7 = 1: The online decoding incorrectly selects the 
Gaussian model, whereas the final decoding correctly selects 
the Gaussian model. 

4. 7 = 2 and 7 = 2: Both the online decoding in Step 2 and the 
final decoding in Step 4 incorrectly select the Gaussian model 
for the signal. 

To further understand the behavior of the four error components, 
Monte Carlo simulation is performed to check them on synthetic 
data. The data set up follows that in |6|, emulating standard behavior 
of image patches: the signals are of dimension N = 6A\ the eigenval- 
ues of the Gaussians follow a power decay law = m~^, I <m< 

with a typical value a = 2; the two Gaussians are "orthogonal" 
one another, i.e., B^B2 = I/r, where Bi and B2 are the PCAs of the 



two Gaussians, and I/^ is the left-right flipped identity matrix. The 
sensing matrix <J> contains M = 16 measurements (sampling rate 1/4), 
and the number K of random measurements varies from 1 to M. The 
Gaussian model selection is more accurate as the two Gaussians are 
orthogonal (see also |2|). On the other hand, when the online model 
selection is erroneous, the resulting principal direction sensing in the 
wrong Gaussian is the farthest possible from optimal. 

Figure [T] plots the four error components (normalized by the sig- 
nal energy) as a function of the number K of random measurements 
in the online SCS. The first component increases as K increases: As 
the online model selection is correctly calculated from the K random 
measurements, M — K principal direction vectors of that Gaussian 
are then used to sense the signal; a larger K leads to a smaller num- 
ber of M — ^ optimal principal direction sensing measurements, and 
thus to a larger error. The second component is constantly zero: If 
the online decoding correctly calculates the model selection from the 
K random measurements, after adding M — K measurements along 
the principal directions in the appropriate Gaussian, the model se- 
lection in the final decoding never goes wrong. The third and the 
fourth components decrease as K increases: The incorrect model 
selection obtained by the online decoding from the first K random 
measurements leads to M — ^ principal direction measurements in 
the wrong Gaussian; in our example, the two Gaussians have the 
opposite eigenvalue order, and these M — K principal direction mea- 
surements in the wrong Gaussian are therefore the worst possible; a 
larger K reduces the number M — ^ of the principal direction mea- 
surements in the wrong Gaussian, and reduces in consequence the 
error, since having more random measurements is better than having 
more of the wrong measurements. 

Figure [2] plots the Gaussian model selection errors of the online 
and final coding as a function of K. Both errors decrease as K in- 
creases. The online model selection error is constantly larger than 
that of the final model selection, the two converging as K goes to M. 

The sum of the four online SCS error components illustrated in 
Figure[T]gives the MSE of the online SCS, plotted in Figure [3] (red) as 
a function of K. The curve presents a U-shape. When K is small, the 
online model selection is inaccurate, the principal direction sensing 
is thus likely in the wrong Gaussian, which results in a large MSE. 
As K increases, the MSE first decreases and then increases, trading 
off between the online model selection accuracy (the larger K is, the 
more accurate the online model selection) and the principal direc- 
tion sensing (the larger K is, the smaller M — K principal direction 
measurements). The MSE of SCS using fully random sensing mea- 
surements is plotted in the same figure (blue) for comparison. The 
lowest point is attained at ^ = 9, where the MSE of the online SCS 
is 0.65 times that of SCS. The online SCS thus considerably reduces 
the MSE of SCS. 

Monte Carlo simulations further show that a similar U-shape 
graph is obtained with different M values and the Gaussian eigen- 
value decay parameter a: The online SCS has the lowest MSE with 
K in the order of M/2, and the ratio between the MSE of the online 
adaptive SCS and the standard SCS is smaller as a and M increase. 

4. EXPERIMENTS WITH REAL IMAGES 

The online adaptive SCS is applied in real image sensing, and com- 
pared with SCS using fully random measurements. The latter has 
been reported to bring about 0.5 to 3.5 dB improvement in PSNR at 
various sample rates with respect to conventional CS based on sparse 
models |6|. 

Following a common practice, an image is decomposed into 
X =8x8 non-overlapping local patches (an image patch 
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Fig. 1. The four error components of the onhne SCS as a function of K, the 
number of the first- step random measurements. 
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Fig. 2. The Gaussian model selection error (%) of the online and final 
coding as a function of K. 

is reshaped and considered as a vector), each regarded as a signal 
and assumed to follow a GMM [TJ. As illustrated in Figure [4j the 
GMM is comprised of / = 19 geometry-motivated Gaussian models, 
each capturing a local direction (see [7] for more details). M = 16 
measurements, or equivalent a sampling rate of M/A/^ = 1/4, are 
applied. The standard images Lena (512 x 512), House (256 x 256), 
and Peppers (512 x 512), as shown in Figure |5] are used in the 
experiments. 

Figure |6] plots the PSNR of the reconstructed patches obtained 
with the proposed online adaptive SCS as a function of K, the num- 
ber of first- step random measurements, in comparison with that of 
standard SCS. Similar to the U-shape curve obtained on the synthetic 
data in Section [3]2j for all the three images under test, as K increases 
the PSNR of the adaptive SCS overall first rises, and then decreases, 
converging to that of the standard SCS as K goes to M. The largest 
improvement with respect to standard SCS, about 1.5 dB, is attained 
at^ = 9or 10. 

5. CONCLUSION AND FUTURE WORKS 

An online adaptive sensing strategy has been developed for signals 
following a mixture model. The basic idea is to first detect online 
the model, and then adapt the sensing for it. Illustrated for GMMs, 
the framework considerably reduces the average reconstruction er- 
ror with respect to standard CS using fully random measurements on 
both synthetic and real image data, at marginally increased complex- 
ity. 

We are currently refining the proposed algorithm. The hard 
switch from random sensing to optimal sensing triggered by the 
online model selection may be improved with a sample-per-sample 
optimization following ([9]), or extending the analysis developed 
inCl. 
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Fig. 4. The first eight principal direction vectors of a directional PC A. 




Fig. 5. From left to right: Lena, House, and Peppers. 

The proposed scheme imbeds low-level pattern recognition 
(model selection) in the signal sensing and estimation problem. 
The pattern recognition part has value by itself, and will be further 
explored. 

Following the recent results in |1|, the same type of adaptive 
sensing strategy can be applied to mixtures of other distributions. 
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Fig. 3. The MSB of online adaptive SCS as a function of K, in comparison 
to that of SCS using fully random sensing matrices. 



Fig. 6. PSNR comparison of the proposed online adaptive SCS and standard 
SCS as a function K, for three images under test. 



